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DETAILED ACTION 
Drawings 

1 . Figures 5 and 6 should be designated by a legend such as -Prior Art- because only that 
which is old is illustrated. See MPEP § 608.02(g). Corrected drawings in compliance with 37 
CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. 
The replacement sheet(s) should be labeled "Replacement Sheet" in the page header (as per 37 
CFR 1 .84(c)) so as not to obstruct any portion of the drawing figures. If the changes are not 
accepted by the examiner, the applicant will be notified and informed of any required corrective 
action in the next Office action. The objection to the drawings will not be held in abeyance. 

2. The drawings are objected to as failing to comply with 37 CFR 1 .84(p)(5) because they 
include the following reference character(s) not mentioned in the description: r2' (Fig. 7), and r3' 
(Fig. 7). Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the 
specification to add the reference character(s) in the description in compliance with 37 CFR 
1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any 
amended replacement drawing sheet should include all of the figures appearing on the immediate 
prior version of the sheet, even if only one figure is being amended. Each drawing sheet 
submitted after the filing date of an application must be labeled in the top margin as either 
"Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not 
accepted by the examiner, the applicant will be notified and informed of any required corrective 
action in the next Office action. The objection to the drawings will not be held in abeyance. 
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Specification 

3. The lengthy specification has not been checked to the extent necessary to determine the 
presence of all possible minor errors. Applicant's cooperation is requested in correcting any 
errors of which applicant may become aware in the specification. 

Claim Rejections - J5 USC § 101 

4. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

5. Claims 6, 10, 11 and 12 are rejected under 35 U.S.C. 101 because the claimed 
invention is directed to non-statutory subject matter. Claims 6, 10, 1 1 and 12 recite a 
computer program not being technologically embodied. See MPEP § 2106 with regard to 
computer programs. To expedite a complete examination of the instant application, the claims 
rejected under 35 U.S.C § 101 as non-statutory subject matter are further rejected as set forth 
below in anticipation of applicant amending the claims to place them within the four categories 
of invention. 

Claim Rejections - 35 USC §103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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7. The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 
(1966), that are applied for establishing a background for determining obviousness under 35 
U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating obviousness 
or nonobviousness. 

8. Claims 1, 6, 7 and 10 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent No. 5,594,850 to Noyama et al (herein referred to as "Noyama et al") in view of 
U.S. Patent No. 6,028,606 to Kolb et al (herein referred to as "Kolb et al"). 

9. With regard to claim 1, Noyama et al discloses "a method for compositing a computer- 
graphics image and a picture taken by a camera (lines 41-44 of column 4: "Some specific objects 
of the invention are: (1) to create a composite image from a CG image and a natural image... ") 
comprising: 

a. defining a three-dimensional model, a viewpoint (lines 26-29 of column 6: "In 
FIG. 1, a CG model 10 consists of an object shape model 12 made up of object shape and 
object surface attributes, a light source model 14 including the light source position and 
light properties, and an eye position 16. "), and a plane of projection, in a space 
established on a computer (lines 29-34 of column 6: "In ordinary computer graphics, the 
aim is to create a CG image 20 projected on a two dimensional plane, which will not be 
described since it is well known. In this invention, however, in addition to data gathering 
by creating the CG image 20 as in the prior art... "); 
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b. tracing the lines of sight extending from the viewpoint through the plane of 
projection and the three-dimensional model to obtain attributes of portions of the three- 
dimensional model corresponding to the projection pixels (line 66 of column 7 through 
line 3 of column 8: "The ray tracing starts from the eye 16 and continues along a straight 
line connecting the eye 16 and the pixel being processed on the projection plane until 
striking an object, e.g. a shape object 12 (104), then a check is made as to whether or not 
the object struck is the light source 14 (106). "), thereby forming a two-dimensional image 
of the three-dimensional model on the plane of projection (lines 39-41 of column 8: " The 
determined pixel color is written to the intrinsic image 28 (126). "); and 

c 
9 

c. superposing (Figure 8) the two-dimensional image (20) on the picture (30) to 
generate a composite image (50) (lines 22-29 of column 10: "Then the simulation section 
300 creates a final composite image by combining the transformation data for only the 
window (not the inside wall) of the memory section 354, on top of the temporary 
composite image. The final composite image 50 displays a region 55 including the object 
66 in front of the interior wall and in the room behind the glass of the window 64. ")." 

10. Noyama et al does not expressly disclose "defining lines of sight extending from the 
viewpoint to projection pixels on the plane of projection so that each of the lines of sight 
conforms with a ray of light incident on a pixel corresponding thereto of the picture taken by the 
camera." 

1 1 . Kolb et al discloses a function to define lines of sight based upon the "calibration data" 
for a camera system, and "defines lines of sight extending from the viewpoint to projection 
pixels on the plane of projection so that each of the lines of sight conforms with a ray of light 
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incident on a pixel corresponding thereto of the picture taken by the camera" (lines 57-59 of 
column 12: "After computing W the ray tracing algorithm is applied to construct (108) a ray 
from x' to x", and then compute (110) the ray from x" to the scene data"; lines 6-9 of column 8: 
"Rather than computing the direction of a ray using the ray tracing procedure directly, ray 
tracing can alternatively be used to define a function that accurately approximates the way in 
which rays are acted upon by the lens system. "). 

12. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to use the physical camera model calibration data as taught by Kolb et al to determine the 
direction of the rays in the method and system for compositing acquired and synthetic images 
disclosed by Noyama et al. The suggestion and motivation for doing so is given by Kolb et al in 
lines 39-49 of column 2: 

For example in many applications (video special effects, augmented reality, etc.) it is 
desirable to seamlessly merge acquired imagery with synthetic imagery... In both of these 
situations it is important that the synthetic imagery be computed using a camera model 
that closely approximates the real camera and lens system. 

Therefore, it would have been obvious to combine Noyama et al with Kolb et al to obtain the 

invention specified in claim 1 . 

13. With regard to claim 6, the program recited is similar in scope to a computer 
implementation of the method of claim 1 ; Noyama et al discloses a computer implementation in 
lines 24-29 of column 13: "As explained in the foregoing, in accordance with the present 
invention images created with computer graphics are automatically and simultaneously 
transformed into transformation data that can be directly handled in image simulation and, 
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therefore, a computer running an image simulation program can be immediately used for 
creating composite images with natural images, " Claim 6 is rejected with the rationale of claim 
1. 

14. With regard to claim 7, Noyama et al discloses "a method for rendering a three- 
dimensional model created by computer graphics into a two-dimensional image to be superposed 
on a picture taken by a camera to form a composite image, the method comprising: 

d. defining a viewpoint (lines 26-29 of column <5), and a plane of projection, in a 
space established on a computer where the three-dimensional model is located (lines 29- 
34 of column 6)\ 

e. defining lines of sight extending from the viewpoint to projection pixels on the 
plane of projection (lines 18-21 of column 9, see rejection of claim /); 

f tracing the lines of sight extending from the viewpoint through the plane of 
projection and the three-dimensional model to obtain attributes of portions of the three- 
dimensional model corresponding to the projection pixels (line 66 of column 7 through 
line 3 of column #); and 

g. setting the obtained attributes of the portions of the three-dimensional model to 
the projection pixels corresponding thereto, to form a two-dimensional image of the 
three-dimensional model on the plane of projection (lines 39-41 of column 8; lines 22-29 
of column 10)" 

15. Noyama et al does not disclose "defining lines of sight extending from the viewpoint to 
projection pixels on the plane of projection so that each of the lines of sight conforms with a ray 
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of light incident on a pixel corresponding thereto of the picture taken by the camera." Kolb et al 
discloses this limitation in lines 57-59 of column 12 as shown in the rejection of claim 1. 

16. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to use the physical camera model calibration data as taught by Kolb et al to determine the 
direction of the rays in the method and system for compositing acquired and synthetic images 
disclosed by Noyama et al. The suggestion and motivation for doing so is given by Kolb et al in 
lines 39-49 of column 2 as shown in the rejection of claim 1. Therefore, it would have been 
obvious to combine Noyama et al with Kolb et al to obtain the invention specified in claim 7. 

17. With regard to claim 10, the program recited is similar in scope to a computer 
implementation of the method of claim 7; Noyama et al discloses a computer implementation in 
lines 24-29 of column 13: "As explained in the foregoing, in accordance with the present 
invention images created with computer graphics are automatically and simultaneously 
transformed into transformation data that can be directly handled in image simulation and, 
therefore, a computer running an image simulation program can be immediately used for 
creating composite images with natural images. " Claim 10 is rejected with the rationale of claim 
7. 

18. Claims 2, 3, 4, 8, 9, 11 and 12 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Noyama et al in view of Kolb et al and in further view of Benjamin 
Mora, Jean Pierre Jessel, Rene Caubet, "A New Object-Order Ray-Casting Algorithm," 
October 27, 2002, Proceedings of the Conference on Visualization f 02 (herein referred to as 
"Mora et al"). 
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19. With regard to claims 2 and 8, Noyama et al discloses the limitations of parent claim 1, 
but does not disclose a calibration table correlating pixel positions and directions of rays of light. 
Kolb et al discloses a function to define lines of sight based upon the "calibration data" for a 
camera system, "wherein the lines of sight are defined based upon the directions and positions of 
the rays of light incident on the pixels of the picture corresponding to the projection pixels" 
{lines 57-59 of column 12). 

20. Kolb et al does not disclose using a table to store and access the rays computed using the 
lens simulation function. Mora et al stores precomputed ray data corresponding to pixel 
positions. Mora et al discloses "providing a table having first data and second data correlated 
with each other (Figure 2), the first data concerning positions of pixels (section 3.1: " Therefore, 
a square made of four neighboring pixels is subdivided and a list of relative coordinates 
corresponding to the projection of the cell is associated with each subdivision (Pixel index in fig. 
2a). "), the second data concerning ray data (section 3.1: " The set of preprocessed rays (fig. 2b) 
is used to find out the ray entry point within the cell... Thus, a ray number pointing to the best 
representative preprocessed ray is assigned to every pixel of the projection lists (ray index in fig 
2a). "). 

21 . At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to modify the system disclosed by Noyama et al as modified by Kolb et al to store the 
pixel to ray correspondences, as well as the directions and positions of rays in a table as taught 
by Mora et al. The motivation for doing so would have been to avoid redundant computation 
during rendering, as this advantage of precomputing and storing the results is well known in the 
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art. Therefore, it would have been obvious to further modify the combination of Noyama et al 
and Kolb et al with Mora et al to obtain the invention specified in claims 2 and 8. 

22. With regard to claims 1 1 and 12, the program recited is similar in scope to a computer 
implementation of the method of claim 2; Noyama et al discloses a computer implementation in 
lines 24-29 of column 13. Claims 1 1 and 12 are rejected with the rationale of claim 2. 

23. With regard to claim 3, Noyama et al discloses "compositing a computer-graphics image 
created by rendering a three-dimensional model and a picture taken by a camera (Figure 8, see 
rejection of claim 7), comprising: 

h. obtaining lines of sight extending from a viewpoint to the three-dimensional 
model (lines 18-21 of column 9, see rejection of claim 7); 

i. generating a two-dimensional image on the plane of projection from the three- 
dimensional model (lines 39-41 of column 8, see rejection of claim 1) by tracing the lines 
of sight so as to obtain attributes of portions of the three-dimensional model 
corresponding to the projection pixels on the plane of projection (line 66 of column 7 
through line 3 of column 8, see rejection of claim 7); 

j. superposing the two-dimensional image on the picture, to generate a composite 
image (lines 22-29 of column 10, see rejection of claim 7)." 

24. Noyama et al does not expressly disclose basing the line of sight calculation on the 
acquired imagery and corresponding camera. With regard to lines 10-12 of claim 3, Kolb et al 
discloses "each of lines of sight passing through projection pixels on a plane of projection 
conforms with a ray of light incident on a pixel corresponding thereto of the picture taken by the 
camera" and basing the line of sight calculations "upon the directions and positions of the rays of 
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light incident on the pixels of the picture" (lines 57-59 of column 12, see rejection of claim 1). 
As shown in the rejection of claim 1, it would have been obvious to one of ordinary skill in the 
art to modify Noyama et al with Kolb et al to model the lines of sight after those of the picture 
taken by the camera to obtain the advantage of "seamlessly merging acquired imagery with 
synthetic imagery" as suggested by Kolb et al. 

25. With regard to the limitations recited in lines 3-6 of and lines 9-10 of claim 3, Noyama et 
al does not disclose a "calibration storage unit" or "looking up directions and positions." The 
limitations recited in lines 3-6 of and lines 8-10 of claim 3 are similar in scope to the limitations 
recited in claim 2. As shown in the rejection of claim 2, it would have been obvious to one of 
ordinary skill in the art to further modify the combination of Noyama et al and Kolb et al with 
the precomputed table disclosed by Mora et al to obtain the advantage of computational 
efficiency. 

26. Noyama et al does not expressly disclose an "apparatus" with "units." At the time of the 
invention, it would have been obvious to a person of ordinary skill in the art to implement the 
functionality of the method taught by the combination of Noyama et al, Kolb et al, and Mora et 
al in hardware. Kolb et al suggests an apparatus in lines 8-11 of column 15: "Additionally, a real 
time virtual camera could be implemented in hardware." The motivation for doing so would 
have been to obtain a performance gain, as suggested by Kolb et al in lines 8-12 of column 15: 
"In this example, one would make a thick lens approximation to the lens system... which allows 
for speedy real time rendering." Therefore, it would have been obvious to combine Noyama et al, 
Kolb et al and Mora et al to obtain the invention specified in claim 3. 
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27. Claim 9 is rejected with the rationale of claim 3. Claim 9 recites a calibration table, a 
line-of-sight calculation unit, and a two-dimensional image generation unit similar in scope to 
the corresponding computational units recited in claim 3. Furthermore, Noyama et al discloses 
"rendering a three-dimensional model created by computer graphics into a two-dimensional 
image to be superposed on a picture taken by a camera to form a composite image" in Figure 8. 

28. With regard to claim 4, parent claim 3 is met by the combination of Noyama et al, Kolb 
et al and Mora et al. Kolb et al further discloses the process of tracing a ray of light which strikes 
an image point x' and a "displacement from a base point" in lines 28-31 of column 10: "A point 
70 of intersection of a ray from x' to S with P' is found, and then translated parallel to the axis to 
a point 74 on P. A ray from 74 through x, the image of x', is then used to sample the scene." 

29. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to further modify the combination of Noyama et al, Kolb et al and Mora et al to store the 
direction in which a ray strikes on a pixel, and a displacement from a base point as disclosed by 
Kolb et al in a table as taught by Mora et al. The motivation for doing so would have been to 
avoid redundant computation of these values during rendering calculations. Therefore, it would 
have been obvious to further modify the combination of Noyama et al, Kolb et al and Mora et al 
to obtain the invention specified in claim 4. 

30. Claim 5 is rejected under 35 U.S.C 103(a) as being unpatentable over Noyama et al 
in view of Kolb et al in further view of Mora et al and in further view in F, S. Hill, Jr. 
"Computer Graphics Using OpenGL," May 15, 2000, 2nd Edition, Prentice Hall (herein 
referred to as "Hill"). 
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3 1 . With regard to claim 5, the combination of Noyama et al, Kolb et al and Mora et al does 
not show storing as "two points on the incident light." On page 148, Hill teaches representing a 
vector as a difference of two points Q and P. On page 147, Hill states "it is valuable to think of a 
vector geometrically as a displacement from one point to another." 

32. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to represent the rays in storage in the system disclosed by the combination of Noyama et 
al, Kolb et al and Mora et al with two points as disclosed by Hill. The motivation for doing so 
would have been to simplify logic and storage by representing direction, magnitude, and spatial 
orientation with two analogous entities. Therefore, it would have been obvious to further modify 
Noyama et al, Kolb et al and Mora et al with Hill to obtain the invention specified in claim 5. 

Conclusion 

33. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. Ned Greene, Paul S. Heckbert, "Creating raster Omnimax Images from Multiple 
Perspective Views Using the Elliptical Weighted Average Filter," June 1986, IEEE Computer 
Graphics and Applications, v.6 n.6, p.21-27 discloses relating a pixel to "a ray through a 
projection lens," (Section 3) and a weight lookup table (Section 5). Nathan A. Carr, Jesse D. 
Hall, John C. Hart, "The Ray Engine," September 1, 2002, Proceedings of the ACM 
SIGGRAPH/EUROGRAPHICS Conference on Graphics Hardware, discloses a mapping of 
pixels to ray directions. U.S. Patent No. 6,268,863 to Rioux discloses a method for rendering 
images based on camera calibration data. 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jason M. Repko whose telephone number is 571-272-8624. The 
examiner can normally be reached on Monday through Friday 8:30 am -5:00 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka Chauhan can be reached on 571-272-7782. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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SUPERVISORY PATENT EXAMINER 



